Learning FRAME Models Using CNN Filters

The convolutional neural network (ConvNet or CNN) has proven to be very successful in many tasks such as those in computer vision. Recently there has been growing interest in visualizing the knowledge discriminatively learned by CNNs, by generating images based on CNN features. This paper is a contribution towards this theme of research on knowledge visualization via image generation. Specifically, we propose to learn the generative FRAME (Filters, Random field, And Maximum Entropy) model using the highly expressive filters pre-learned by the CNN at the convolutional layers. We show that the learning algorithm can generate vivid images, and we explain that each learned model corresponds to a CNN unit at a layer above the layer of filters employed by the model. Such a CNN-FRAME generative model is different from existing visualization methods, and the proposed learning method enables us to add CNN nodes by learning from small numbers of training examples in a generative fashion.